Search for: All records

BackgroundMetamodels can address some of the limitations of complex simulation models by formulating a mathematical relationship between input parameters and simulation model outcomes. Our objective was to develop and compare the performance of a machine learning (ML)–based metamodel against a conventional metamodeling approach in replicating the findings of a complex simulation model. MethodsWe constructed 3 ML-based metamodels using random forest, support vector regression, and artificial neural networks and a linear regression-based metamodel from a previously validated microsimulation model of the natural history hepatitis C virus (HCV) consisting of 40 input parameters. Outcomes of interest included societal costs and quality-adjusted life-years (QALYs), the incremental cost-effectiveness (ICER) of HCV treatment versus no treatment, cost-effectiveness analysis curve (CEAC), and expected value of perfect information (EVPI). We evaluated metamodel performance using root mean squared error (RMSE) and Pearson’s R²on the normalized data. ResultsThe R²values for the linear regression metamodel for QALYs without treatment, QALYs with treatment, societal cost without treatment, societal cost with treatment, and ICER were 0.92, 0.98, 0.85, 0.92, and 0.60, respectively. The corresponding R²values for our ML-based metamodels were 0.96, 0.97, 0.90, 0.95, and 0.49 for support vector regression; 0.99, 0.83, 0.99, 0.99, and 0.82 for artificial neural network; and 0.99, 0.99, 0.99, 0.99, and 0.98 for random forest. Similar trends were observed for RMSE. The CEAC and EVPI curves produced by the random forest metamodel matched the results of the simulation output more closely than the linear regression metamodel. ConclusionsML-based metamodels generally outperformed traditional linear regression metamodels at replicating results from complex simulation models, with random forest metamodels performing best. HighlightsDecision-analytic models are frequently used by policy makers and other stakeholders to assess the impact of new medical technologies and interventions. However, complex models can impose limitations on conducting probabilistic sensitivity analysis and value-of-information analysis, and may not be suitable for developing online decision-support tools. Metamodels, which accurately formulate a mathematical relationship between input parameters and model outcomes, can replicate complex simulation models and address the above limitation. The machine learning–based random forest model can outperform linear regression in replicating the findings of a complex simulation model. Such a metamodel can be used for conducting cost-effectiveness and value-of-information analyses or developing online decision support tools. 

Abstract Hepatitis C virus (HCV) is 15 times more prevalent among persons in Spain’s prisons than in the community. Recently, Spain initiated a pilot program, JAILFREE-C, to treat HCV in prisons using direct-acting antivirals (DAAs). Our aim was to identify a cost-effective strategy to scale-up HCV treatment in all prisons. Using a validated agent-based model, we simulated the HCV landscape in Spain’s prisons considering disease transmission, screening, treatment, and prison-community dynamics. Costs and disease outcomes under status quo were compared with strategies to scale-up treatment in prisons considering prioritization (HCV fibrosis stage vs. HCV prevalence of prisons), treatment capacity (2,000/year vs. unlimited) and treatment initiation based on sentence lengths (>6 months vs. any). Scaling-up treatment by treating all incarcerated persons irrespective of their sentence length provided maximum health benefits–preventing 10,200 new cases of HCV, and 8,300 HCV-related deaths between 2019–2050; 90% deaths prevented would have occurred in the community. Compared with status quo, this strategy increased quality-adjusted life year (QALYs) by 69,700 and costs by €670 million, yielding an incremental cost-effectiveness ratio of €9,600/QALY. Scaling-up HCV treatment with DAAs for the entire Spanish prison population, irrespective of sentence length, is cost-effective and would reduce HCV burden. 

Objectives Oral direct-acting antivirals (DAAs) for hepatitis C virus (HCV) have dramatically changed the treatment paradigm. Our aim was to project temporal trends in HCV diagnosis, treatment and disease burden in France, Germany, Italy, Spain and the UK. Design A mathematical simulation model of natural history of HCV infection. Participants HCV-infected patients defined based on country-specific age, fibrosis and genotype distributions. Interventions HCV screening practice and availability of different waves of DAA treatment in each country. Outcome measures Temporal trends in the number of patients who achieve sustained virological response (SVR), fail treatment (by drug regimen) and develop advanced sequelae from 2014 to 2030 in each country. Results We projected that 1 324 000 individuals would receive treatment from 2014 to 2030 in the five European countries and 12 000–37 000 of them would fail to achieve SVR. By 2021, the number of individuals cured of HCV would supersede the number of actively infected individuals in France, Germany, Spain and the UK. Under status quo, the diagnosis rate would reach between 65% and 75% and treatment coverage between 65% and 74% by 2030 in these countries. The number of patients who fail treatment would decrease over time, with the majority of those who fail treatment having been exposed to non-structural protein 5A inhibitors. Conclusions In the era of DAAs, the number of people with HCV who achieved a cure will exceed the number of viraemic patients, but many patients will remain undiagnosed, untreated, fail multiple treatments and develop advanced sequelae. Scaling-up screening and treatment capacity, and timely and effective retreatment are needed to avail the full benefits of DAAs and to meet HCV elimination targets set by WHO. 

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1–4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.